---

title: Optuna: A hyperparameter optimization framework


keywords: fastai
sidebar: home_sidebar

summary: "Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters."
description: "Optuna is an automatic hyperparameter optimization software framework, particularly designed for machine learning. It features an imperative, define-by-run style user API. Thanks to our define-by-run API, the code written with Optuna enjoys high modularity, and the user of Optuna can dynamically construct the search spaces for the hyperparameters."
nb_path: "nbs/200_optuna.ipynb"
---
<!--

#################################################
### THIS FILE WAS AUTOGENERATED! DO NOT EDIT! ###
#################################################
# file to edit: nbs/200_optuna.ipynb
# command to build the docs after a change: nbdev_build_docs

-->

<div class="container" id="notebook-container">
        
    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">

<div class="output_wrapper">
<div class="output">

<div class="output_area">


<div class="output_markdown rendered_html output_subarea ">
<h4 id="run_optuna_study" class="doc_header"><code>run_optuna_study</code><a href="https://github.com/timeseriesAI/tsai/tree/main/tsai/optuna.py#L15" class="source_link" style="float:right">[source]</a></h4><blockquote><p><code>run_optuna_study</code>(<strong><code>objective</code></strong>, <strong><code>resume</code></strong>=<em><code>None</code></em>, <strong><code>study_type</code></strong>=<em><code>None</code></em>, <strong><code>multivariate</code></strong>=<em><code>True</code></em>, <strong><code>search_space</code></strong>=<em><code>None</code></em>, <strong><code>evaluate</code></strong>=<em><code>None</code></em>, <strong><code>seed</code></strong>=<em><code>None</code></em>, <strong><code>sampler</code></strong>=<em><code>None</code></em>, <strong><code>pruner</code></strong>=<em><code>None</code></em>, <strong><code>study_name</code></strong>=<em><code>None</code></em>, <strong><code>direction</code></strong>=<em><code>'maximize'</code></em>, <strong><code>load_if_exists</code></strong>=<em><code>False</code></em>, <strong><code>n_trials</code></strong>=<em><code>None</code></em>, <strong><code>timeout</code></strong>=<em><code>None</code></em>, <strong><code>gc_after_trial</code></strong>=<em><code>False</code></em>, <strong><code>show_progress_bar</code></strong>=<em><code>True</code></em>, <strong><code>save_study</code></strong>=<em><code>True</code></em>, <strong><code>path</code></strong>=<em><code>'optuna'</code></em>, <strong><code>show_plots</code></strong>=<em><code>True</code></em>)</p>
</blockquote>
<p>Creates and runs an optuna study.</p>
<p>Args:
    objective:          A callable that implements objective function.
    resume:             Path to a previously saved study.
    study_type:         Type of study selected (bayesian, gridsearch, randomsearch). Based on this a sampler will be build if sampler is None.
                        If a sampler is passed, this has no effect.
    multivariate:       If this is True, the multivariate TPE is used when suggesting parameters. The multivariate TPE is reported to outperform
                        the independent TPE.
    search_space:       Search space required when running a gridsearch (if you don't pass a sampler).
    evaluate:           Allows you to pass a specific set of hyperparameters that will be evaluated.
    seed:               Fixed seed used by samplers.
    sampler:            A sampler object that implements background algorithm for value suggestion. If None is specified, TPESampler is used during
                        single-objective optimization and NSGAIISampler during multi-objective optimization. See also samplers.
    pruner:             A pruner object that decides early stopping of unpromising trials. If None is specified, MedianPruner is used as the default.
                        See also pruners.
    study_name:         Study’s name. If this argument is set to None, a unique name is generated automatically.
    direction:          A sequence of directions during multi-objective optimization.
    n_trials:           The number of trials. If this argument is set to None, there is no limitation on the number of trials. If timeout is also set to
                        None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM.
    timeout:            Stop study after the given number of second(s). If this argument is set to None, the study is executed without time limitation.
                        If n_trials is also set to None, the study continues to create trials until it receives a termination signal such as
                        Ctrl+C or SIGTERM.
    gc_after_trial:     Flag to execute garbage collection at the end of each trial. By default, garbage collection is enabled, just in case.
                        You can turn it off with this argument if memory is safely managed in your objective function.
    show_progress_bar:  Flag to show progress bars or not. To disable progress bar, set this False.
    save_study:         Save your study when finished/ interrupted.
    path:               Folder where the study will be saved.
    show_plots:         Flag to control whether plots are shown at the end of the study.</p>

</div>

</div>

</div>
</div>

</div>
    {% endraw %}

    {% raw %}
    
<div class="cell border-box-sizing code_cell rendered">
<div class="input">

<div class="inner_cell">
    <div class="input_area">
<div class=" highlight hl-ipython3"><pre><span></span><span class="k">def</span> <span class="nf">run_optuna_study</span><span class="p">(</span><span class="n">objective</span><span class="p">,</span> <span class="n">resume</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">study_type</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">multivariate</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">search_space</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">evaluate</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">seed</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">sampler</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">pruner</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">study_name</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> 
                     <span class="n">direction</span><span class="o">=</span><span class="s1">&#39;maximize&#39;</span><span class="p">,</span> <span class="n">load_if_exists</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">n_trials</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="kc">None</span><span class="p">,</span> <span class="n">gc_after_trial</span><span class="o">=</span><span class="kc">False</span><span class="p">,</span> <span class="n">show_progress_bar</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">save_study</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> 
                     <span class="n">path</span><span class="o">=</span><span class="s1">&#39;optuna&#39;</span><span class="p">,</span> <span class="n">show_plots</span><span class="o">=</span><span class="kc">True</span><span class="p">):</span>
    <span class="sa">r</span><span class="sd">&quot;&quot;&quot;Creates and runs an optuna study.</span>

<span class="sd">    Args: </span>
<span class="sd">        objective:          A callable that implements objective function.</span>
<span class="sd">        resume:             Path to a previously saved study.</span>
<span class="sd">        study_type:         Type of study selected (bayesian, gridsearch, randomsearch). Based on this a sampler will be build if sampler is None. </span>
<span class="sd">                            If a sampler is passed, this has no effect.</span>
<span class="sd">        multivariate:       If this is True, the multivariate TPE is used when suggesting parameters. The multivariate TPE is reported to outperform </span>
<span class="sd">                            the independent TPE.</span>
<span class="sd">        search_space:       Search space required when running a gridsearch (if you don&#39;t pass a sampler).</span>
<span class="sd">        evaluate:           Allows you to pass a specific set of hyperparameters that will be evaluated.</span>
<span class="sd">        seed:               Fixed seed used by samplers.</span>
<span class="sd">        sampler:            A sampler object that implements background algorithm for value suggestion. If None is specified, TPESampler is used during </span>
<span class="sd">                            single-objective optimization and NSGAIISampler during multi-objective optimization. See also samplers.</span>
<span class="sd">        pruner:             A pruner object that decides early stopping of unpromising trials. If None is specified, MedianPruner is used as the default. </span>
<span class="sd">                            See also pruners.</span>
<span class="sd">        study_name:         Study’s name. If this argument is set to None, a unique name is generated automatically.</span>
<span class="sd">        direction:          A sequence of directions during multi-objective optimization.</span>
<span class="sd">        n_trials:           The number of trials. If this argument is set to None, there is no limitation on the number of trials. If timeout is also set to </span>
<span class="sd">                            None, the study continues to create trials until it receives a termination signal such as Ctrl+C or SIGTERM.</span>
<span class="sd">        timeout:            Stop study after the given number of second(s). If this argument is set to None, the study is executed without time limitation. </span>
<span class="sd">                            If n_trials is also set to None, the study continues to create trials until it receives a termination signal such as </span>
<span class="sd">                            Ctrl+C or SIGTERM.</span>
<span class="sd">        gc_after_trial:     Flag to execute garbage collection at the end of each trial. By default, garbage collection is enabled, just in case. </span>
<span class="sd">                            You can turn it off with this argument if memory is safely managed in your objective function.</span>
<span class="sd">        show_progress_bar:  Flag to show progress bars or not. To disable progress bar, set this False.</span>
<span class="sd">        save_study:         Save your study when finished/ interrupted.</span>
<span class="sd">        path:               Folder where the study will be saved.</span>
<span class="sd">        show_plots:         Flag to control whether plots are shown at the end of the study.</span>
<span class="sd">    &quot;&quot;&quot;</span>
    
    <span class="k">try</span><span class="p">:</span> <span class="kn">import</span> <span class="nn">optuna</span>
    <span class="k">except</span> <span class="ne">ImportError</span><span class="p">:</span> <span class="k">raise</span> <span class="ne">ImportError</span><span class="p">(</span><span class="s1">&#39;You need to install optuna!&#39;</span><span class="p">)</span>

    <span class="c1"># Sampler</span>
    <span class="k">if</span> <span class="n">sampler</span> <span class="ow">is</span> <span class="kc">None</span><span class="p">:</span>
        <span class="k">if</span> <span class="n">study_type</span> <span class="ow">is</span> <span class="kc">None</span> <span class="ow">or</span> <span class="s2">&quot;bayes&quot;</span> <span class="ow">in</span> <span class="n">study_type</span><span class="o">.</span><span class="n">lower</span><span class="p">():</span> 
            <span class="n">sampler</span> <span class="o">=</span> <span class="n">optuna</span><span class="o">.</span><span class="n">samplers</span><span class="o">.</span><span class="n">TPESampler</span><span class="p">(</span><span class="n">seed</span><span class="o">=</span><span class="n">seed</span><span class="p">,</span> <span class="n">multivariate</span><span class="o">=</span><span class="n">multivariate</span><span class="p">)</span>
        <span class="k">elif</span> <span class="s2">&quot;grid&quot;</span> <span class="ow">in</span> <span class="n">study_type</span><span class="o">.</span><span class="n">lower</span><span class="p">():</span>
            <span class="k">assert</span> <span class="n">search_space</span><span class="p">,</span> <span class="sa">f</span><span class="s2">&quot;you need to pass a search_space dict to run a gridsearch&quot;</span>
            <span class="n">sampler</span> <span class="o">=</span> <span class="n">optuna</span><span class="o">.</span><span class="n">samplers</span><span class="o">.</span><span class="n">GridSampler</span><span class="p">(</span><span class="n">search_space</span><span class="p">)</span>
        <span class="k">elif</span> <span class="s2">&quot;random&quot;</span> <span class="ow">in</span> <span class="n">study_type</span><span class="o">.</span><span class="n">lower</span><span class="p">():</span> 
            <span class="n">sampler</span> <span class="o">=</span> <span class="n">optuna</span><span class="o">.</span><span class="n">samplers</span><span class="o">.</span><span class="n">RandomSampler</span><span class="p">(</span><span class="n">seed</span><span class="o">=</span><span class="n">seed</span><span class="p">)</span>
    <span class="k">assert</span> <span class="n">sampler</span><span class="p">,</span> <span class="s2">&quot;you need to either select a study type (bayesian, gridsampler, randomsampler) or pass a sampler&quot;</span>

    <span class="c1"># Study</span>
    <span class="k">if</span> <span class="n">resume</span><span class="p">:</span> 
        <span class="k">try</span><span class="p">:</span>
            <span class="n">study</span> <span class="o">=</span> <span class="n">joblib</span><span class="o">.</span><span class="n">load</span><span class="p">(</span><span class="n">resume</span><span class="p">)</span>
        <span class="k">except</span><span class="p">:</span> 
            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;joblib.load(</span><span class="si">{</span><span class="n">resume</span><span class="si">}</span><span class="s2">) couldn&#39;t recover any saved study. Check the path.&quot;</span><span class="p">)</span>
            <span class="k">return</span>
        <span class="nb">print</span><span class="p">(</span><span class="s2">&quot;Best trial until now:&quot;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="s2">&quot; Value: &quot;</span><span class="p">,</span> <span class="n">study</span><span class="o">.</span><span class="n">best_trial</span><span class="o">.</span><span class="n">value</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="s2">&quot; Params: &quot;</span><span class="p">)</span>
        <span class="k">for</span> <span class="n">key</span><span class="p">,</span> <span class="n">value</span> <span class="ow">in</span> <span class="n">study</span><span class="o">.</span><span class="n">best_trial</span><span class="o">.</span><span class="n">params</span><span class="o">.</span><span class="n">items</span><span class="p">():</span>
            <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;    </span><span class="si">{</span><span class="n">key</span><span class="si">}</span><span class="s2">: </span><span class="si">{</span><span class="n">value</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
    <span class="k">else</span><span class="p">:</span> 
        <span class="n">study</span> <span class="o">=</span> <span class="n">optuna</span><span class="o">.</span><span class="n">create_study</span><span class="p">(</span><span class="n">sampler</span><span class="o">=</span><span class="n">sampler</span><span class="p">,</span> <span class="n">pruner</span><span class="o">=</span><span class="n">pruner</span><span class="p">,</span> <span class="n">study_name</span><span class="o">=</span><span class="n">study_name</span><span class="p">,</span> <span class="n">direction</span><span class="o">=</span><span class="n">direction</span><span class="p">)</span>
    <span class="k">if</span> <span class="n">evaluate</span><span class="p">:</span> <span class="n">study</span><span class="o">.</span><span class="n">enqueue_trial</span><span class="p">(</span><span class="n">evaluate</span><span class="p">)</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">study</span><span class="o">.</span><span class="n">optimize</span><span class="p">(</span><span class="n">objective</span><span class="p">,</span> <span class="n">n_trials</span><span class="o">=</span><span class="n">n_trials</span><span class="p">,</span> <span class="n">timeout</span><span class="o">=</span><span class="n">timeout</span><span class="p">,</span> <span class="n">gc_after_trial</span><span class="o">=</span><span class="n">gc_after_trial</span><span class="p">,</span> <span class="n">show_progress_bar</span><span class="o">=</span><span class="n">show_progress_bar</span><span class="p">)</span>
    <span class="k">except</span> <span class="ne">KeyboardInterrupt</span><span class="p">:</span>
        <span class="k">pass</span>

    <span class="c1"># Save</span>
    <span class="k">if</span> <span class="n">save_study</span><span class="p">:</span>
        <span class="n">full_path</span> <span class="o">=</span> <span class="n">Path</span><span class="p">(</span><span class="n">path</span><span class="p">)</span><span class="o">/</span><span class="sa">f</span><span class="s1">&#39;</span><span class="si">{</span><span class="n">study</span><span class="o">.</span><span class="n">study_name</span><span class="si">}</span><span class="s1">.pkl&#39;</span>
        <span class="n">full_path</span><span class="o">.</span><span class="n">parent</span><span class="o">.</span><span class="n">mkdir</span><span class="p">(</span><span class="n">parents</span><span class="o">=</span><span class="kc">True</span><span class="p">,</span> <span class="n">exist_ok</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
        <span class="n">joblib</span><span class="o">.</span><span class="n">dump</span><span class="p">(</span><span class="n">study</span><span class="p">,</span> <span class="n">full_path</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">Optuna study saved to </span><span class="si">{</span><span class="n">full_path</span><span class="si">}</span><span class="s1">&#39;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;To reload the study run: study = joblib.load(&#39;</span><span class="si">{</span><span class="n">full_path</span><span class="si">}</span><span class="s2">&#39;)&quot;</span><span class="p">)</span>

    <span class="c1"># Plots</span>
    <span class="k">if</span> <span class="n">show_plots</span> <span class="ow">and</span> <span class="nb">len</span><span class="p">(</span><span class="n">study</span><span class="o">.</span><span class="n">trials</span><span class="p">)</span> <span class="o">&gt;</span> <span class="mi">1</span><span class="p">:</span>
        <span class="k">try</span><span class="p">:</span> <span class="n">display</span><span class="p">(</span><span class="n">optuna</span><span class="o">.</span><span class="n">visualization</span><span class="o">.</span><span class="n">plot_optimization_history</span><span class="p">(</span><span class="n">study</span><span class="p">))</span>
        <span class="k">except</span><span class="p">:</span> <span class="k">pass</span>
        <span class="k">try</span><span class="p">:</span> <span class="n">display</span><span class="p">(</span><span class="n">optuna</span><span class="o">.</span><span class="n">visualization</span><span class="o">.</span><span class="n">plot_param_importances</span><span class="p">(</span><span class="n">study</span><span class="p">))</span>
        <span class="k">except</span><span class="p">:</span> <span class="k">pass</span>
        <span class="k">try</span><span class="p">:</span> <span class="n">display</span><span class="p">(</span><span class="n">optuna</span><span class="o">.</span><span class="n">visualization</span><span class="o">.</span><span class="n">plot_slice</span><span class="p">(</span><span class="n">study</span><span class="p">))</span>
        <span class="k">except</span><span class="p">:</span> <span class="k">pass</span>
        <span class="k">try</span><span class="p">:</span> <span class="n">display</span><span class="p">(</span><span class="n">optuna</span><span class="o">.</span><span class="n">visualization</span><span class="o">.</span><span class="n">plot_parallel_coordinate</span><span class="p">(</span><span class="n">study</span><span class="p">))</span>
        <span class="k">except</span><span class="p">:</span> <span class="k">pass</span>

    <span class="c1"># Study stats</span>
    <span class="k">try</span><span class="p">:</span>
        <span class="n">pruned_trials</span> <span class="o">=</span> <span class="p">[</span><span class="n">t</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">study</span><span class="o">.</span><span class="n">trials</span> <span class="k">if</span> <span class="n">t</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="n">optuna</span><span class="o">.</span><span class="n">trial</span><span class="o">.</span><span class="n">TrialState</span><span class="o">.</span><span class="n">PRUNED</span><span class="p">]</span>
        <span class="n">complete_trials</span> <span class="o">=</span> <span class="p">[</span><span class="n">t</span> <span class="k">for</span> <span class="n">t</span> <span class="ow">in</span> <span class="n">study</span><span class="o">.</span><span class="n">trials</span> <span class="k">if</span> <span class="n">t</span><span class="o">.</span><span class="n">state</span> <span class="o">==</span> <span class="n">optuna</span><span class="o">.</span><span class="n">trial</span><span class="o">.</span><span class="n">TrialState</span><span class="o">.</span><span class="n">COMPLETE</span><span class="p">]</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="se">\n</span><span class="s2">Study statistics    : &quot;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  Study name        : </span><span class="si">{</span><span class="n">study</span><span class="o">.</span><span class="n">study_name</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  # finished trials : </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">study</span><span class="o">.</span><span class="n">trials</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  # pruned trials   : </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">pruned_trials</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  # complete trials : </span><span class="si">{</span><span class="nb">len</span><span class="p">(</span><span class="n">complete_trials</span><span class="p">)</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
        
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;</span><span class="se">\n</span><span class="s2">Best trial          :&quot;</span><span class="p">)</span>
        <span class="n">trial</span> <span class="o">=</span> <span class="n">study</span><span class="o">.</span><span class="n">best_trial</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  value             : </span><span class="si">{</span><span class="n">trial</span><span class="o">.</span><span class="n">value</span><span class="si">}</span><span class="s2">&quot;</span><span class="p">)</span>
        <span class="nb">print</span><span class="p">(</span><span class="sa">f</span><span class="s2">&quot;  best_params = </span><span class="si">{</span><span class="n">trial</span><span class="o">.</span><span class="n">params</span><span class="si">}</span><span class="se">\n</span><span class="s2">&quot;</span><span class="p">)</span>
    <span class="k">except</span><span class="p">:</span>
        <span class="nb">print</span><span class="p">(</span><span class="s1">&#39;</span><span class="se">\n</span><span class="s1">No finished trials yet.&#39;</span><span class="p">)</span>
    <span class="k">return</span> <span class="n">study</span>
</pre></div>

    </div>
</div>
</div>

</div>
    {% endraw %}

</div>
 

